{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.1"},"colab":{"name":"1-basic-rnn.ipynb","provenance":[]}},"cells":[{"cell_type":"markdown","metadata":{"id":"n4dNIlHtzJa6"},"source":["# Basic RNN\n","- Objective: to understand basics of RNN & LSTM"]},{"cell_type":"markdown","metadata":{"id":"2XjGSyD0zJa8"},"source":["## Recurrent Neural Networks\n","- Feedforward neural networks (e.g. MLPs and CNNs) are powerful, but they are not optimized to handle \"sequential\" data\n","- In other words, they do not possess \"memory\" of previous inputs\n","- For instance, consider the case of translating a corpus. You need to consider the **\"context\"** to guess the next word to come forward\n","\n","<img src=\"http://2.bp.blogspot.com/-9GIdV292xV4/UwOIy6B6koI/AAAAAAAAHi4/X6UGlyHI-_U/s1600/tumblr_ms5qzpFY671r9nm7io1_500.gif\" style=\"width: 500px\"/>\n","\n","<br>\n","- RNNs are suitable for dealing with sequential format data since they have **\"recurrent\"** structure\n","- To put it differently, they keep the **\"memory\"** of earlier inputs in the sequence\n","</br>\n","<img src=\"http://www.wildml.com/wp-content/uploads/2015/09/rnn.jpg\" style=\"width: 600px\"/>\n","\n","<br>\n","- However, in order to reduce the number of parameters, every layer of different time steps shares same parameters\n","</br>\n","\n","<img src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png\" style=\"width: 600px\"/>"]},{"cell_type":"markdown","metadata":{"id":"_2SD0BuyzJa8"},"source":["## Load Dataset"]},{"cell_type":"code","metadata":{"id":"g_rD7drqzJa9","executionInfo":{"status":"ok","timestamp":1604888360129,"user_tz":420,"elapsed":2959,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["import numpy as np\n","\n","from sklearn.metrics import accuracy_score\n","from tensorflow.keras.datasets import reuters\n","from tensorflow.keras.preprocessing.sequence import pad_sequences\n","from tensorflow.keras.utils import to_categorical"],"execution_count":1,"outputs":[]},{"cell_type":"code","metadata":{"id":"4hZppYUzzJbA","executionInfo":{"status":"ok","timestamp":1604888361955,"user_tz":420,"elapsed":695,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["# parameters for data load\n","num_words = 30000\n","maxlen = 50\n","test_split = 0.3"],"execution_count":2,"outputs":[]},{"cell_type":"code","metadata":{"id":"AmS9LN4TzJbD","executionInfo":{"status":"ok","timestamp":1604888362996,"user_tz":420,"elapsed":1143,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"487a0cea-c552-4750-a06b-f2da9e4c5719","colab":{"base_uri":"https://localhost:8080/"}},"source":["(X_train, y_train), (X_test, y_test) = reuters.load_data(num_words = num_words, maxlen = maxlen, test_split = test_split)"],"execution_count":3,"outputs":[{"output_type":"stream","text":["Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/reuters.npz\n","2113536/2110848 [==============================] - 0s 0us/step\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"0e843OUOzJbF","executionInfo":{"status":"ok","timestamp":1604888364309,"user_tz":420,"elapsed":598,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["# pad the sequences with zeros \n","# padding parameter is set to 'post' => 0's are appended to end of sequences\n","X_train = pad_sequences(X_train, padding = 'post')\n","X_test = pad_sequences(X_test, padding = 'post')"],"execution_count":4,"outputs":[]},{"cell_type":"code","metadata":{"id":"gDw_7bOjzJbI","executionInfo":{"status":"ok","timestamp":1604888365085,"user_tz":420,"elapsed":474,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["X_train = np.array(X_train).reshape((X_train.shape[0], X_train.shape[1], 1))\n","X_test = np.array(X_test).reshape((X_test.shape[0], X_test.shape[1], 1))"],"execution_count":5,"outputs":[]},{"cell_type":"code","metadata":{"id":"lhN_j4F-zJbL","executionInfo":{"status":"ok","timestamp":1604888365768,"user_tz":420,"elapsed":466,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["y_data = np.concatenate((y_train, y_test))\n","y_data = to_categorical(y_data)"],"execution_count":6,"outputs":[]},{"cell_type":"code","metadata":{"id":"9aziJ943zJbP","executionInfo":{"status":"ok","timestamp":1604888366527,"user_tz":420,"elapsed":220,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["y_train = y_data[:1395]\n","y_test = y_data[1395:]"],"execution_count":7,"outputs":[]},{"cell_type":"code","metadata":{"id":"HinpbuzHzJbR","executionInfo":{"status":"ok","timestamp":1604888367214,"user_tz":420,"elapsed":345,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"70ac55c5-cb5c-4ed4-efca-8bfc6a400318","colab":{"base_uri":"https://localhost:8080/"}},"source":["print(X_train.shape)\n","print(X_test.shape)\n","print(y_train.shape)\n","print(y_test.shape)"],"execution_count":8,"outputs":[{"output_type":"stream","text":["(1395, 49, 1)\n","(599, 49, 1)\n","(1395, 46)\n","(599, 46)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"W5-mzgBozJbU"},"source":["## 1. Vanilla RNN\n","- Vanilla RNNs have a simple structure\n","- However, they suffer from the problem of \"long-term dependencies\"\n","- Hence, they are not able to keep the **sequential memory\" for long\n","\n","<img src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-SimpleRNN.png\" style=\"width: 600px\"/>"]},{"cell_type":"code","metadata":{"id":"po2XO6q6zJbV","executionInfo":{"status":"ok","timestamp":1604888374023,"user_tz":420,"elapsed":325,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["from tensorflow.keras.models import Sequential\n","from tensorflow.keras.layers import Dense, SimpleRNN, Activation\n","from tensorflow.keras import optimizers\n","from tensorflow.keras.wrappers.scikit_learn import KerasClassifier"],"execution_count":9,"outputs":[]},{"cell_type":"code","metadata":{"id":"P1hZj7eDzJbX","executionInfo":{"status":"ok","timestamp":1604888375121,"user_tz":420,"elapsed":447,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["def vanilla_rnn():\n","    model = Sequential()\n","    model.add(SimpleRNN(50, input_shape = (49,1), return_sequences = False))\n","    model.add(Dense(46))\n","    model.add(Activation('softmax'))\n","    \n","    adam = optimizers.Adam(lr = 0.001)\n","    model.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['accuracy'])\n","    \n","    return model"],"execution_count":10,"outputs":[]},{"cell_type":"code","metadata":{"id":"ui3Zl90ozJba","executionInfo":{"status":"ok","timestamp":1604888376431,"user_tz":420,"elapsed":408,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["model = KerasClassifier(build_fn = vanilla_rnn, epochs = 200, batch_size = 50, verbose = 1)"],"execution_count":11,"outputs":[]},{"cell_type":"code","metadata":{"id":"t4oHmOUSzJbc","executionInfo":{"status":"ok","timestamp":1604888431594,"user_tz":420,"elapsed":55122,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"afbaf548-97a3-4d3e-a962-74d9a31dad88","colab":{"base_uri":"https://localhost:8080/"}},"source":["model.fit(X_train, y_train)"],"execution_count":12,"outputs":[{"output_type":"stream","text":["Epoch 1/200\n","28/28 [==============================] - 0s 8ms/step - loss: 3.1467 - accuracy: 0.2158\n","Epoch 2/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.5311 - accuracy: 0.6789\n","Epoch 3/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.2370 - accuracy: 0.7054\n","Epoch 4/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1850 - accuracy: 0.7147\n","Epoch 5/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1672 - accuracy: 0.7147\n","Epoch 6/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1637 - accuracy: 0.7147\n","Epoch 7/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1581 - accuracy: 0.7147\n","Epoch 8/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1537 - accuracy: 0.7147\n","Epoch 9/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1519 - accuracy: 0.7147\n","Epoch 10/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1513 - accuracy: 0.7147\n","Epoch 11/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1491 - accuracy: 0.7147\n","Epoch 12/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1471 - accuracy: 0.7147\n","Epoch 13/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1447 - accuracy: 0.7147\n","Epoch 14/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1448 - accuracy: 0.7147\n","Epoch 15/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1428 - accuracy: 0.7147\n","Epoch 16/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1410 - accuracy: 0.7147\n","Epoch 17/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1424 - accuracy: 0.7161\n","Epoch 18/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1399 - accuracy: 0.7147\n","Epoch 19/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1391 - accuracy: 0.7161\n","Epoch 20/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1360 - accuracy: 0.7190\n","Epoch 21/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1368 - accuracy: 0.7161\n","Epoch 22/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1327 - accuracy: 0.7197\n","Epoch 23/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1331 - accuracy: 0.7204\n","Epoch 24/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1311 - accuracy: 0.7190\n","Epoch 25/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1307 - accuracy: 0.7183\n","Epoch 26/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1340 - accuracy: 0.7176\n","Epoch 27/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.1195 - accuracy: 0.7190\n","Epoch 28/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1121 - accuracy: 0.7190\n","Epoch 29/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1257 - accuracy: 0.7154\n","Epoch 30/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1193 - accuracy: 0.7211\n","Epoch 31/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0976 - accuracy: 0.7197\n","Epoch 32/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.0692 - accuracy: 0.7226\n","Epoch 33/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.1092 - accuracy: 0.7168\n","Epoch 34/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.0973 - accuracy: 0.7176\n","Epoch 35/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0809 - accuracy: 0.7204\n","Epoch 36/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0678 - accuracy: 0.7269\n","Epoch 37/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0539 - accuracy: 0.7183\n","Epoch 38/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.0585 - accuracy: 0.7118\n","Epoch 39/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0347 - accuracy: 0.7247\n","Epoch 40/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0502 - accuracy: 0.7254\n","Epoch 41/200\n","28/28 [==============================] - 0s 8ms/step - loss: 1.0335 - accuracy: 0.7254\n","Epoch 42/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0428 - accuracy: 0.7254\n","Epoch 43/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0155 - accuracy: 0.7240\n","Epoch 44/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0165 - accuracy: 0.7240\n","Epoch 45/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9972 - accuracy: 0.7312\n","Epoch 46/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0061 - accuracy: 0.7290\n","Epoch 47/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9924 - accuracy: 0.7362\n","Epoch 48/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9742 - accuracy: 0.7348\n","Epoch 49/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9921 - accuracy: 0.7326\n","Epoch 50/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0024 - accuracy: 0.7219\n","Epoch 51/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9781 - accuracy: 0.7254\n","Epoch 52/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9651 - accuracy: 0.7326\n","Epoch 53/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9560 - accuracy: 0.7319\n","Epoch 54/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9641 - accuracy: 0.7362\n","Epoch 55/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9786 - accuracy: 0.7348\n","Epoch 56/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9536 - accuracy: 0.7305\n","Epoch 57/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9469 - accuracy: 0.7419\n","Epoch 58/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9457 - accuracy: 0.7419\n","Epoch 59/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9393 - accuracy: 0.7376\n","Epoch 60/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9547 - accuracy: 0.7290\n","Epoch 61/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9624 - accuracy: 0.7276\n","Epoch 62/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0096 - accuracy: 0.7168\n","Epoch 63/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9557 - accuracy: 0.7305\n","Epoch 64/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9177 - accuracy: 0.7398\n","Epoch 65/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.9301 - accuracy: 0.7419\n","Epoch 66/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9211 - accuracy: 0.7405\n","Epoch 67/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9024 - accuracy: 0.7405\n","Epoch 68/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9038 - accuracy: 0.7391\n","Epoch 69/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9057 - accuracy: 0.7441\n","Epoch 70/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8983 - accuracy: 0.7398\n","Epoch 71/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8855 - accuracy: 0.7441\n","Epoch 72/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8828 - accuracy: 0.7470\n","Epoch 73/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8903 - accuracy: 0.7484\n","Epoch 74/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8844 - accuracy: 0.7498\n","Epoch 75/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9720 - accuracy: 0.7462\n","Epoch 76/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9154 - accuracy: 0.7455\n","Epoch 77/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.8846 - accuracy: 0.7477\n","Epoch 78/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9003 - accuracy: 0.7477\n","Epoch 79/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8888 - accuracy: 0.7491\n","Epoch 80/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.8873 - accuracy: 0.7462\n","Epoch 81/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8698 - accuracy: 0.7455\n","Epoch 82/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8735 - accuracy: 0.7427\n","Epoch 83/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8653 - accuracy: 0.7513\n","Epoch 84/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8619 - accuracy: 0.7513\n","Epoch 85/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8649 - accuracy: 0.7491\n","Epoch 86/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8523 - accuracy: 0.7527\n","Epoch 87/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9075 - accuracy: 0.7441\n","Epoch 88/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.9210 - accuracy: 0.7283\n","Epoch 89/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9004 - accuracy: 0.7455\n","Epoch 90/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9700 - accuracy: 0.7168\n","Epoch 91/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0831 - accuracy: 0.6946\n","Epoch 92/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0231 - accuracy: 0.7082\n","Epoch 93/200\n","28/28 [==============================] - 0s 9ms/step - loss: 1.0060 - accuracy: 0.7133\n","Epoch 94/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9946 - accuracy: 0.7240\n","Epoch 95/200\n","28/28 [==============================] - 0s 8ms/step - loss: 0.9851 - accuracy: 0.7219\n","Epoch 96/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9517 - accuracy: 0.7233\n","Epoch 97/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9309 - accuracy: 0.7290\n","Epoch 98/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9211 - accuracy: 0.7341\n","Epoch 99/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9018 - accuracy: 0.7434\n","Epoch 100/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8895 - accuracy: 0.7455\n","Epoch 101/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8995 - accuracy: 0.7419\n","Epoch 102/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8818 - accuracy: 0.7412\n","Epoch 103/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8639 - accuracy: 0.7462\n","Epoch 104/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8578 - accuracy: 0.7477\n","Epoch 105/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8481 - accuracy: 0.7470\n","Epoch 106/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8518 - accuracy: 0.7491\n","Epoch 107/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8528 - accuracy: 0.7541\n","Epoch 108/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8430 - accuracy: 0.7513\n","Epoch 109/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8372 - accuracy: 0.7541\n","Epoch 110/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8340 - accuracy: 0.7556\n","Epoch 111/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8367 - accuracy: 0.7527\n","Epoch 112/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8456 - accuracy: 0.7534\n","Epoch 113/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8672 - accuracy: 0.7427\n","Epoch 114/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8457 - accuracy: 0.7470\n","Epoch 115/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8407 - accuracy: 0.7527\n","Epoch 116/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8432 - accuracy: 0.7534\n","Epoch 117/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8307 - accuracy: 0.7534\n","Epoch 118/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8225 - accuracy: 0.7577\n","Epoch 119/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8316 - accuracy: 0.7534\n","Epoch 120/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8235 - accuracy: 0.7548\n","Epoch 121/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8176 - accuracy: 0.7563\n","Epoch 122/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8200 - accuracy: 0.7556\n","Epoch 123/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8141 - accuracy: 0.7527\n","Epoch 124/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8174 - accuracy: 0.7513\n","Epoch 125/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8153 - accuracy: 0.7548\n","Epoch 126/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8115 - accuracy: 0.7505\n","Epoch 127/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8053 - accuracy: 0.7563\n","Epoch 128/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8137 - accuracy: 0.7491\n","Epoch 129/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8184 - accuracy: 0.7527\n","Epoch 130/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8128 - accuracy: 0.7548\n","Epoch 131/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.9044 - accuracy: 0.7434\n","Epoch 132/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8454 - accuracy: 0.7520\n","Epoch 133/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8558 - accuracy: 0.7455\n","Epoch 134/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8310 - accuracy: 0.7534\n","Epoch 135/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8149 - accuracy: 0.7577\n","Epoch 136/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8361 - accuracy: 0.7441\n","Epoch 137/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8986 - accuracy: 0.7226\n","Epoch 138/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8792 - accuracy: 0.7376\n","Epoch 139/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8498 - accuracy: 0.7470\n","Epoch 140/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8230 - accuracy: 0.7541\n","Epoch 141/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.8553 - accuracy: 0.7419\n","Epoch 142/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8290 - accuracy: 0.7534\n","Epoch 143/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8144 - accuracy: 0.7534\n","Epoch 144/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8139 - accuracy: 0.7527\n","Epoch 145/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8038 - accuracy: 0.7505\n","Epoch 146/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8164 - accuracy: 0.7513\n","Epoch 147/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.8258 - accuracy: 0.7491\n","Epoch 148/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8081 - accuracy: 0.7548\n","Epoch 149/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7997 - accuracy: 0.7548\n","Epoch 150/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7974 - accuracy: 0.7556\n","Epoch 151/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7994 - accuracy: 0.7563\n","Epoch 152/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7984 - accuracy: 0.7541\n","Epoch 153/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7989 - accuracy: 0.7541\n","Epoch 154/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7950 - accuracy: 0.7541\n","Epoch 155/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7936 - accuracy: 0.7563\n","Epoch 156/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7892 - accuracy: 0.7606\n","Epoch 157/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7927 - accuracy: 0.7534\n","Epoch 158/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7862 - accuracy: 0.7591\n","Epoch 159/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7890 - accuracy: 0.7534\n","Epoch 160/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7838 - accuracy: 0.7606\n","Epoch 161/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7943 - accuracy: 0.7520\n","Epoch 162/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7828 - accuracy: 0.7591\n","Epoch 163/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7808 - accuracy: 0.7570\n","Epoch 164/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7823 - accuracy: 0.7548\n","Epoch 165/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7804 - accuracy: 0.7570\n","Epoch 166/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7758 - accuracy: 0.7563\n","Epoch 167/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7729 - accuracy: 0.7620\n","Epoch 168/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7783 - accuracy: 0.7548\n","Epoch 169/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7730 - accuracy: 0.7570\n","Epoch 170/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7756 - accuracy: 0.7606\n","Epoch 171/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7714 - accuracy: 0.7606\n","Epoch 172/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7649 - accuracy: 0.7591\n","Epoch 173/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7816 - accuracy: 0.7563\n","Epoch 174/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7770 - accuracy: 0.7606\n","Epoch 175/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7755 - accuracy: 0.7570\n","Epoch 176/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7987 - accuracy: 0.7563\n","Epoch 177/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7762 - accuracy: 0.7591\n","Epoch 178/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7747 - accuracy: 0.7556\n","Epoch 179/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7702 - accuracy: 0.7606\n","Epoch 180/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7624 - accuracy: 0.7556\n","Epoch 181/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7663 - accuracy: 0.7563\n","Epoch 182/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7556 - accuracy: 0.7606\n","Epoch 183/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7539 - accuracy: 0.7663\n","Epoch 184/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7522 - accuracy: 0.7677\n","Epoch 185/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7543 - accuracy: 0.7656\n","Epoch 186/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7585 - accuracy: 0.7591\n","Epoch 187/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7546 - accuracy: 0.7634\n","Epoch 188/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7554 - accuracy: 0.7606\n","Epoch 189/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7633 - accuracy: 0.7563\n","Epoch 190/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7541 - accuracy: 0.7620\n","Epoch 191/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7584 - accuracy: 0.7649\n","Epoch 192/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7478 - accuracy: 0.7627\n","Epoch 193/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7434 - accuracy: 0.7677\n","Epoch 194/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7562 - accuracy: 0.7620\n","Epoch 195/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7440 - accuracy: 0.7670\n","Epoch 196/200\n","28/28 [==============================] - 0s 10ms/step - loss: 0.7776 - accuracy: 0.7570\n","Epoch 197/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.8144 - accuracy: 0.7541\n","Epoch 198/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7866 - accuracy: 0.7577\n","Epoch 199/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7548 - accuracy: 0.7584\n","Epoch 200/200\n","28/28 [==============================] - 0s 9ms/step - loss: 0.7422 - accuracy: 0.7642\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7fbe4cc79d68>"]},"metadata":{"tags":[]},"execution_count":12}]},{"cell_type":"code","metadata":{"id":"39bP0HQzzJbf","executionInfo":{"status":"ok","timestamp":1604888431766,"user_tz":420,"elapsed":54835,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"93cab192-0e92-48d1-d78c-f59f76f1835f","colab":{"base_uri":"https://localhost:8080/"}},"source":["y_pred = model.predict(X_test)"],"execution_count":13,"outputs":[{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/wrappers/scikit_learn.py:241: Sequential.predict_classes (from tensorflow.python.keras.engine.sequential) is deprecated and will be removed after 2021-01-01.\n","Instructions for updating:\n","Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype(\"int32\")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).\n","12/12 [==============================] - 0s 3ms/step\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"rT5wCslSzJbh","executionInfo":{"status":"ok","timestamp":1604888431768,"user_tz":420,"elapsed":54207,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["y_test_ = np.argmax(y_test, axis = 1)"],"execution_count":14,"outputs":[]},{"cell_type":"code","metadata":{"id":"i7Ec0NjMzJbk","executionInfo":{"status":"ok","timestamp":1604888431769,"user_tz":420,"elapsed":53698,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"b7c63640-9425-46ea-805b-b5fe11ef21a7","colab":{"base_uri":"https://localhost:8080/"}},"source":["print(accuracy_score(y_pred, y_test_))"],"execution_count":15,"outputs":[{"output_type":"stream","text":["0.7495826377295493\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"H_2mWjLlzJbm"},"source":["## 2. Stacked Vanilla RNN\n","- RNN layers can be stacked to form a deeper network\n","\n","<img src=\"https://lh6.googleusercontent.com/rC1DSgjlmobtRxMPFi14hkMdDqSkEkuOX7EW_QrLFSymjasIM95Za2Wf-VwSC1Tq1sjJlOPLJ92q7PTKJh2hjBoXQawM6MQC27east67GFDklTalljlt0cFLZnPMdhp8erzO\" style=\"width: 500px\"/>"]},{"cell_type":"code","metadata":{"id":"0L8o9VESzJbm","executionInfo":{"status":"ok","timestamp":1604888432243,"user_tz":420,"elapsed":472,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["def stacked_vanilla_rnn():\n","    model = Sequential()\n","    model.add(SimpleRNN(50, input_shape = (49,1), return_sequences = True))   # return_sequences parameter has to be set True to stack\n","    model.add(SimpleRNN(50, return_sequences = False))\n","    model.add(Dense(46))\n","    model.add(Activation('softmax'))\n","    \n","    adam = optimizers.Adam(lr = 0.001)\n","    model.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['accuracy'])\n","    \n","    return model"],"execution_count":16,"outputs":[]},{"cell_type":"code","metadata":{"id":"AXyC_tPfzJbp","executionInfo":{"status":"ok","timestamp":1604888432246,"user_tz":420,"elapsed":430,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["model = KerasClassifier(build_fn = stacked_vanilla_rnn, epochs = 200, batch_size = 50, verbose = 1)"],"execution_count":17,"outputs":[]},{"cell_type":"code","metadata":{"id":"vCqeckrgzJbr","executionInfo":{"status":"ok","timestamp":1604888545704,"user_tz":420,"elapsed":113884,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"eae1a33b-6974-4082-f186-6f32c285951f","colab":{"base_uri":"https://localhost:8080/"}},"source":["model.fit(X_train, y_train)"],"execution_count":18,"outputs":[{"output_type":"stream","text":["Epoch 1/200\n","28/28 [==============================] - 1s 19ms/step - loss: 2.4446 - accuracy: 0.5333\n","Epoch 2/200\n","28/28 [==============================] - 1s 20ms/step - loss: 1.2726 - accuracy: 0.7147\n","Epoch 3/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.1797 - accuracy: 0.7147\n","Epoch 4/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.1434 - accuracy: 0.7147\n","Epoch 5/200\n","28/28 [==============================] - 1s 18ms/step - loss: 1.1186 - accuracy: 0.7147\n","Epoch 6/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0987 - accuracy: 0.7147\n","Epoch 7/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0930 - accuracy: 0.7147\n","Epoch 8/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0803 - accuracy: 0.7147\n","Epoch 9/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0779 - accuracy: 0.7154\n","Epoch 10/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0730 - accuracy: 0.7147\n","Epoch 11/200\n","28/28 [==============================] - 1s 20ms/step - loss: 1.0669 - accuracy: 0.7140\n","Epoch 12/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0685 - accuracy: 0.7147\n","Epoch 13/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0603 - accuracy: 0.7147\n","Epoch 14/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0661 - accuracy: 0.7154\n","Epoch 15/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0530 - accuracy: 0.7133\n","Epoch 16/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0432 - accuracy: 0.7140\n","Epoch 17/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0420 - accuracy: 0.7140\n","Epoch 18/200\n","28/28 [==============================] - 1s 18ms/step - loss: 1.0379 - accuracy: 0.7168\n","Epoch 19/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0250 - accuracy: 0.7211\n","Epoch 20/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0384 - accuracy: 0.7176\n","Epoch 21/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0403 - accuracy: 0.7154\n","Epoch 22/200\n","28/28 [==============================] - 1s 20ms/step - loss: 1.0323 - accuracy: 0.7147\n","Epoch 23/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0277 - accuracy: 0.7161\n","Epoch 24/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0188 - accuracy: 0.7168\n","Epoch 25/200\n","28/28 [==============================] - 1s 19ms/step - loss: 1.0170 - accuracy: 0.7168\n","Epoch 26/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.9992 - accuracy: 0.7176\n","Epoch 27/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.9919 - accuracy: 0.7204\n","Epoch 28/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.9815 - accuracy: 0.7154\n","Epoch 29/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.9654 - accuracy: 0.7254\n","Epoch 30/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.9813 - accuracy: 0.7183\n","Epoch 31/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.9617 - accuracy: 0.7276\n","Epoch 32/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.9454 - accuracy: 0.7262\n","Epoch 33/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.9348 - accuracy: 0.7183\n","Epoch 34/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.9400 - accuracy: 0.7269\n","Epoch 35/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.9256 - accuracy: 0.7355\n","Epoch 36/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.9254 - accuracy: 0.7254\n","Epoch 37/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.9012 - accuracy: 0.7319\n","Epoch 38/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.9071 - accuracy: 0.7348\n","Epoch 39/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8922 - accuracy: 0.7369\n","Epoch 40/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8850 - accuracy: 0.7341\n","Epoch 41/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8829 - accuracy: 0.7312\n","Epoch 42/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.8692 - accuracy: 0.7369\n","Epoch 43/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8683 - accuracy: 0.7376\n","Epoch 44/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8563 - accuracy: 0.7376\n","Epoch 45/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8597 - accuracy: 0.7384\n","Epoch 46/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8544 - accuracy: 0.7355\n","Epoch 47/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.8399 - accuracy: 0.7462\n","Epoch 48/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8443 - accuracy: 0.7448\n","Epoch 49/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8411 - accuracy: 0.7405\n","Epoch 50/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8339 - accuracy: 0.7434\n","Epoch 51/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.8464 - accuracy: 0.7362\n","Epoch 52/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8158 - accuracy: 0.7462\n","Epoch 53/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.8206 - accuracy: 0.7484\n","Epoch 54/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.8094 - accuracy: 0.7455\n","Epoch 55/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8024 - accuracy: 0.7484\n","Epoch 56/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.7940 - accuracy: 0.7527\n","Epoch 57/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7946 - accuracy: 0.7563\n","Epoch 58/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7979 - accuracy: 0.7455\n","Epoch 59/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7908 - accuracy: 0.7520\n","Epoch 60/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7925 - accuracy: 0.7448\n","Epoch 61/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.7782 - accuracy: 0.7498\n","Epoch 62/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7778 - accuracy: 0.7541\n","Epoch 63/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.7708 - accuracy: 0.7599\n","Epoch 64/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.7602 - accuracy: 0.7620\n","Epoch 65/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7599 - accuracy: 0.7548\n","Epoch 66/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7565 - accuracy: 0.7677\n","Epoch 67/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7521 - accuracy: 0.7599\n","Epoch 68/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7392 - accuracy: 0.7706\n","Epoch 69/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7334 - accuracy: 0.7699\n","Epoch 70/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7388 - accuracy: 0.7670\n","Epoch 71/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7064 - accuracy: 0.7821\n","Epoch 72/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7730 - accuracy: 0.7484\n","Epoch 73/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7817 - accuracy: 0.7477\n","Epoch 74/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.7786 - accuracy: 0.7455\n","Epoch 75/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7689 - accuracy: 0.7448\n","Epoch 76/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7610 - accuracy: 0.7441\n","Epoch 77/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7512 - accuracy: 0.7534\n","Epoch 78/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.7436 - accuracy: 0.7606\n","Epoch 79/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.7385 - accuracy: 0.7599\n","Epoch 80/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7394 - accuracy: 0.7577\n","Epoch 81/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.7238 - accuracy: 0.7613\n","Epoch 82/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7286 - accuracy: 0.7563\n","Epoch 83/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7212 - accuracy: 0.7606\n","Epoch 84/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.6999 - accuracy: 0.7778\n","Epoch 85/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7116 - accuracy: 0.7742\n","Epoch 86/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.8164 - accuracy: 0.7312\n","Epoch 87/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7491 - accuracy: 0.7505\n","Epoch 88/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7192 - accuracy: 0.7634\n","Epoch 89/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.7123 - accuracy: 0.7570\n","Epoch 90/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.7012 - accuracy: 0.7692\n","Epoch 91/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.6946 - accuracy: 0.7663\n","Epoch 92/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.6985 - accuracy: 0.7613\n","Epoch 93/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6939 - accuracy: 0.7599\n","Epoch 94/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6865 - accuracy: 0.7713\n","Epoch 95/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6770 - accuracy: 0.7649\n","Epoch 96/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6781 - accuracy: 0.7728\n","Epoch 97/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6607 - accuracy: 0.7778\n","Epoch 98/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6379 - accuracy: 0.7864\n","Epoch 99/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6434 - accuracy: 0.8000\n","Epoch 100/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6402 - accuracy: 0.7900\n","Epoch 101/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6245 - accuracy: 0.8036\n","Epoch 102/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6470 - accuracy: 0.7935\n","Epoch 103/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.7245 - accuracy: 0.7534\n","Epoch 104/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7119 - accuracy: 0.7534\n","Epoch 105/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.6721 - accuracy: 0.7685\n","Epoch 106/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6497 - accuracy: 0.7821\n","Epoch 107/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.6423 - accuracy: 0.7971\n","Epoch 108/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6702 - accuracy: 0.7857\n","Epoch 109/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6233 - accuracy: 0.7978\n","Epoch 110/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.6799 - accuracy: 0.7677\n","Epoch 111/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6568 - accuracy: 0.7713\n","Epoch 112/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.7292 - accuracy: 0.7642\n","Epoch 113/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.7127 - accuracy: 0.7613\n","Epoch 114/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6872 - accuracy: 0.7706\n","Epoch 115/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.6695 - accuracy: 0.7627\n","Epoch 116/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.6628 - accuracy: 0.7692\n","Epoch 117/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6504 - accuracy: 0.7785\n","Epoch 118/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6575 - accuracy: 0.7735\n","Epoch 119/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6597 - accuracy: 0.7685\n","Epoch 120/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6418 - accuracy: 0.7792\n","Epoch 121/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6440 - accuracy: 0.7749\n","Epoch 122/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6397 - accuracy: 0.7799\n","Epoch 123/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6399 - accuracy: 0.7735\n","Epoch 124/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6365 - accuracy: 0.7763\n","Epoch 125/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6331 - accuracy: 0.7778\n","Epoch 126/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6260 - accuracy: 0.7728\n","Epoch 127/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6228 - accuracy: 0.7792\n","Epoch 128/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6227 - accuracy: 0.7763\n","Epoch 129/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6406 - accuracy: 0.7742\n","Epoch 130/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6211 - accuracy: 0.7842\n","Epoch 131/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.6183 - accuracy: 0.7785\n","Epoch 132/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6141 - accuracy: 0.7857\n","Epoch 133/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6243 - accuracy: 0.7742\n","Epoch 134/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6350 - accuracy: 0.7799\n","Epoch 135/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.6056 - accuracy: 0.7828\n","Epoch 136/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6029 - accuracy: 0.7828\n","Epoch 137/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6005 - accuracy: 0.7864\n","Epoch 138/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.6100 - accuracy: 0.7864\n","Epoch 139/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.6227 - accuracy: 0.7756\n","Epoch 140/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6024 - accuracy: 0.7849\n","Epoch 141/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6058 - accuracy: 0.7821\n","Epoch 142/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6080 - accuracy: 0.7900\n","Epoch 143/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6063 - accuracy: 0.7842\n","Epoch 144/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6081 - accuracy: 0.7756\n","Epoch 145/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6101 - accuracy: 0.7907\n","Epoch 146/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5987 - accuracy: 0.7864\n","Epoch 147/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6254 - accuracy: 0.7763\n","Epoch 148/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6141 - accuracy: 0.7814\n","Epoch 149/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5831 - accuracy: 0.7885\n","Epoch 150/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5800 - accuracy: 0.7907\n","Epoch 151/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5811 - accuracy: 0.7921\n","Epoch 152/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5834 - accuracy: 0.7914\n","Epoch 153/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5856 - accuracy: 0.7950\n","Epoch 154/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5735 - accuracy: 0.7907\n","Epoch 155/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5722 - accuracy: 0.7928\n","Epoch 156/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5556 - accuracy: 0.7957\n","Epoch 157/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5553 - accuracy: 0.7950\n","Epoch 158/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5509 - accuracy: 0.7950\n","Epoch 159/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5678 - accuracy: 0.7921\n","Epoch 160/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5654 - accuracy: 0.7935\n","Epoch 161/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5711 - accuracy: 0.7964\n","Epoch 162/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5547 - accuracy: 0.7950\n","Epoch 163/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5533 - accuracy: 0.8022\n","Epoch 164/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5326 - accuracy: 0.8057\n","Epoch 165/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5137 - accuracy: 0.8294\n","Epoch 166/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5537 - accuracy: 0.8072\n","Epoch 167/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5280 - accuracy: 0.8258\n","Epoch 168/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6275 - accuracy: 0.7814\n","Epoch 169/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.6948 - accuracy: 0.7563\n","Epoch 170/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6195 - accuracy: 0.7835\n","Epoch 171/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.6075 - accuracy: 0.7749\n","Epoch 172/200\n","28/28 [==============================] - 1s 18ms/step - loss: 0.5944 - accuracy: 0.7835\n","Epoch 173/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5789 - accuracy: 0.7878\n","Epoch 174/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5751 - accuracy: 0.7878\n","Epoch 175/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5727 - accuracy: 0.7885\n","Epoch 176/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5646 - accuracy: 0.7907\n","Epoch 177/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5545 - accuracy: 0.7957\n","Epoch 178/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5532 - accuracy: 0.7957\n","Epoch 179/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5481 - accuracy: 0.7935\n","Epoch 180/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5381 - accuracy: 0.8029\n","Epoch 181/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5344 - accuracy: 0.8115\n","Epoch 182/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5918 - accuracy: 0.7799\n","Epoch 183/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5748 - accuracy: 0.7878\n","Epoch 184/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5514 - accuracy: 0.7907\n","Epoch 185/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5503 - accuracy: 0.7907\n","Epoch 186/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5354 - accuracy: 0.7943\n","Epoch 187/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5139 - accuracy: 0.8036\n","Epoch 188/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5179 - accuracy: 0.8115\n","Epoch 189/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5025 - accuracy: 0.8100\n","Epoch 190/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.4958 - accuracy: 0.8158\n","Epoch 191/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5451 - accuracy: 0.7964\n","Epoch 192/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5225 - accuracy: 0.8000\n","Epoch 193/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5179 - accuracy: 0.8036\n","Epoch 194/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5086 - accuracy: 0.8036\n","Epoch 195/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.5083 - accuracy: 0.8086\n","Epoch 196/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.4616 - accuracy: 0.8330\n","Epoch 197/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.4556 - accuracy: 0.8430\n","Epoch 198/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.4587 - accuracy: 0.8409\n","Epoch 199/200\n","28/28 [==============================] - 1s 20ms/step - loss: 0.6221 - accuracy: 0.7821\n","Epoch 200/200\n","28/28 [==============================] - 1s 19ms/step - loss: 0.5389 - accuracy: 0.8136\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7fbe4d608828>"]},"metadata":{"tags":[]},"execution_count":18}]},{"cell_type":"code","metadata":{"id":"AaKxqlcizJbu","executionInfo":{"status":"ok","timestamp":1604888546008,"user_tz":420,"elapsed":114176,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"d6fff989-b241-4c14-87ff-c1c9660e0f11","colab":{"base_uri":"https://localhost:8080/"}},"source":["y_pred = model.predict(X_test)"],"execution_count":19,"outputs":[{"output_type":"stream","text":["12/12 [==============================] - 0s 6ms/step\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"i7p5bZt5zJbw","executionInfo":{"status":"ok","timestamp":1604888546009,"user_tz":420,"elapsed":114166,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"ffb320de-44eb-4d6f-c23e-2e1f76fa1151","colab":{"base_uri":"https://localhost:8080/"}},"source":["print(accuracy_score(y_pred, y_test_))"],"execution_count":20,"outputs":[{"output_type":"stream","text":["0.7746243739565943\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"eZobLnzdzJb1"},"source":["## 3. LSTM\n","- LSTM (long short-term memory) is an improved structure to solve the problem of long-term dependencies\n","\n","<img src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-chain.png\" style=\"width: 600px\"/>"]},{"cell_type":"code","metadata":{"id":"LnV8mAdGzJb2","executionInfo":{"status":"ok","timestamp":1604888546438,"user_tz":420,"elapsed":425,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["from tensorflow.keras.layers import LSTM"],"execution_count":21,"outputs":[]},{"cell_type":"code","metadata":{"id":"PwzGZ2vjzJb4","executionInfo":{"status":"ok","timestamp":1604888546571,"user_tz":420,"elapsed":545,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["def lstm():\n","    model = Sequential()\n","    model.add(LSTM(50, input_shape = (49,1), return_sequences = False))\n","    model.add(Dense(46))\n","    model.add(Activation('softmax'))\n","    \n","    adam = optimizers.Adam(lr = 0.001)\n","    model.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['accuracy'])\n","    \n","    return model"],"execution_count":22,"outputs":[]},{"cell_type":"code","metadata":{"id":"s4GJrIdJzJcC","executionInfo":{"status":"ok","timestamp":1604888546573,"user_tz":420,"elapsed":544,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["model = KerasClassifier(build_fn = lstm, epochs = 200, batch_size = 50, verbose = 1)"],"execution_count":23,"outputs":[]},{"cell_type":"code","metadata":{"id":"BktDBXVezJcF","executionInfo":{"status":"ok","timestamp":1604888680548,"user_tz":420,"elapsed":134515,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"96e930d7-a6cd-4197-a937-de1c23ac9b9a","colab":{"base_uri":"https://localhost:8080/"}},"source":["model.fit(X_train, y_train)"],"execution_count":24,"outputs":[{"output_type":"stream","text":["Epoch 1/200\n","28/28 [==============================] - 1s 21ms/step - loss: 3.2411 - accuracy: 0.6029\n","Epoch 2/200\n","28/28 [==============================] - 1s 22ms/step - loss: 1.4156 - accuracy: 0.7147\n","Epoch 3/200\n","28/28 [==============================] - 1s 22ms/step - loss: 1.1957 - accuracy: 0.7147\n","Epoch 4/200\n","28/28 [==============================] - 1s 23ms/step - loss: 1.1572 - accuracy: 0.7147\n","Epoch 5/200\n","28/28 [==============================] - 1s 22ms/step - loss: 1.1355 - accuracy: 0.7147\n","Epoch 6/200\n","28/28 [==============================] - 1s 22ms/step - loss: 1.1212 - accuracy: 0.7147\n","Epoch 7/200\n","28/28 [==============================] - 1s 22ms/step - loss: 1.1070 - accuracy: 0.7147\n","Epoch 8/200\n","28/28 [==============================] - 1s 22ms/step - loss: 1.0964 - accuracy: 0.7147\n","Epoch 9/200\n","28/28 [==============================] - 1s 22ms/step - loss: 1.0866 - accuracy: 0.7147\n","Epoch 10/200\n","28/28 [==============================] - 1s 22ms/step - loss: 1.0800 - accuracy: 0.7147\n","Epoch 11/200\n","28/28 [==============================] - 1s 23ms/step - loss: 1.0778 - accuracy: 0.7147\n","Epoch 12/200\n","28/28 [==============================] - 1s 22ms/step - loss: 1.0256 - accuracy: 0.7147\n","Epoch 13/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.9832 - accuracy: 0.7627\n","Epoch 14/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.9314 - accuracy: 0.7792\n","Epoch 15/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.8858 - accuracy: 0.7864\n","Epoch 16/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.8493 - accuracy: 0.7950\n","Epoch 17/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.8372 - accuracy: 0.8007\n","Epoch 18/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.8349 - accuracy: 0.7928\n","Epoch 19/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.8308 - accuracy: 0.7957\n","Epoch 20/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.8206 - accuracy: 0.8007\n","Epoch 21/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.8096 - accuracy: 0.8072\n","Epoch 22/200\n","28/28 [==============================] - 1s 21ms/step - loss: 0.8067 - accuracy: 0.8036\n","Epoch 23/200\n","28/28 [==============================] - 1s 21ms/step - loss: 0.7965 - accuracy: 0.8065\n","Epoch 24/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.8047 - accuracy: 0.8036\n","Epoch 25/200\n","28/28 [==============================] - 1s 21ms/step - loss: 0.7931 - accuracy: 0.8086\n","Epoch 26/200\n","28/28 [==============================] - 1s 21ms/step - loss: 0.7853 - accuracy: 0.8129\n","Epoch 27/200\n","28/28 [==============================] - 1s 21ms/step - loss: 0.7707 - accuracy: 0.8136\n","Epoch 28/200\n","28/28 [==============================] - 1s 21ms/step - loss: 0.7801 - accuracy: 0.8179\n","Epoch 29/200\n","28/28 [==============================] - 1s 21ms/step - loss: 0.7713 - accuracy: 0.8115\n","Epoch 30/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7651 - accuracy: 0.8136\n","Epoch 31/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7584 - accuracy: 0.8172\n","Epoch 32/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7561 - accuracy: 0.8136\n","Epoch 33/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7437 - accuracy: 0.8215\n","Epoch 34/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7357 - accuracy: 0.8165\n","Epoch 35/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.7203 - accuracy: 0.8287\n","Epoch 36/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7271 - accuracy: 0.8244\n","Epoch 37/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7735 - accuracy: 0.8093\n","Epoch 38/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7782 - accuracy: 0.8129\n","Epoch 39/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.7500 - accuracy: 0.8143\n","Epoch 40/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7435 - accuracy: 0.8208\n","Epoch 41/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7283 - accuracy: 0.8215\n","Epoch 42/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.7218 - accuracy: 0.8258\n","Epoch 43/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.7342 - accuracy: 0.8179\n","Epoch 44/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.7114 - accuracy: 0.8272\n","Epoch 45/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6988 - accuracy: 0.8280\n","Epoch 46/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.7141 - accuracy: 0.8194\n","Epoch 47/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.8473 - accuracy: 0.7993\n","Epoch 48/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.7738 - accuracy: 0.8208\n","Epoch 49/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.7311 - accuracy: 0.8237\n","Epoch 50/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.7116 - accuracy: 0.8308\n","Epoch 51/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6989 - accuracy: 0.8315\n","Epoch 52/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.7003 - accuracy: 0.8351\n","Epoch 53/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6976 - accuracy: 0.8308\n","Epoch 54/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6796 - accuracy: 0.8358\n","Epoch 55/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6778 - accuracy: 0.8358\n","Epoch 56/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6878 - accuracy: 0.8287\n","Epoch 57/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6728 - accuracy: 0.8387\n","Epoch 58/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6751 - accuracy: 0.8394\n","Epoch 59/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6702 - accuracy: 0.8387\n","Epoch 60/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6623 - accuracy: 0.8401\n","Epoch 61/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6706 - accuracy: 0.8344\n","Epoch 62/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6635 - accuracy: 0.8380\n","Epoch 63/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6619 - accuracy: 0.8401\n","Epoch 64/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6659 - accuracy: 0.8394\n","Epoch 65/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6530 - accuracy: 0.8401\n","Epoch 66/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6574 - accuracy: 0.8380\n","Epoch 67/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6642 - accuracy: 0.8373\n","Epoch 68/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6441 - accuracy: 0.8437\n","Epoch 69/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6340 - accuracy: 0.8487\n","Epoch 70/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6478 - accuracy: 0.8409\n","Epoch 71/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6272 - accuracy: 0.8466\n","Epoch 72/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6554 - accuracy: 0.8366\n","Epoch 73/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6677 - accuracy: 0.8387\n","Epoch 74/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6630 - accuracy: 0.8323\n","Epoch 75/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6562 - accuracy: 0.8358\n","Epoch 76/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6511 - accuracy: 0.8380\n","Epoch 77/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6457 - accuracy: 0.8387\n","Epoch 78/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6440 - accuracy: 0.8409\n","Epoch 79/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6320 - accuracy: 0.8401\n","Epoch 80/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6198 - accuracy: 0.8437\n","Epoch 81/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6352 - accuracy: 0.8430\n","Epoch 82/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6295 - accuracy: 0.8394\n","Epoch 83/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6179 - accuracy: 0.8452\n","Epoch 84/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6133 - accuracy: 0.8495\n","Epoch 85/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5993 - accuracy: 0.8466\n","Epoch 86/200\n","28/28 [==============================] - 1s 24ms/step - loss: 0.6015 - accuracy: 0.8473\n","Epoch 87/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6242 - accuracy: 0.8401\n","Epoch 88/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6067 - accuracy: 0.8480\n","Epoch 89/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.6044 - accuracy: 0.8444\n","Epoch 90/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5945 - accuracy: 0.8480\n","Epoch 91/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5739 - accuracy: 0.8530\n","Epoch 92/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5859 - accuracy: 0.8502\n","Epoch 93/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5966 - accuracy: 0.8502\n","Epoch 94/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.6374 - accuracy: 0.8401\n","Epoch 95/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5977 - accuracy: 0.8459\n","Epoch 96/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5953 - accuracy: 0.8437\n","Epoch 97/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5812 - accuracy: 0.8502\n","Epoch 98/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5969 - accuracy: 0.8401\n","Epoch 99/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5644 - accuracy: 0.8552\n","Epoch 100/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5513 - accuracy: 0.8595\n","Epoch 101/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5517 - accuracy: 0.8573\n","Epoch 102/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5663 - accuracy: 0.8495\n","Epoch 103/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5601 - accuracy: 0.8566\n","Epoch 104/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5718 - accuracy: 0.8509\n","Epoch 105/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5661 - accuracy: 0.8495\n","Epoch 106/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5483 - accuracy: 0.8573\n","Epoch 107/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5376 - accuracy: 0.8595\n","Epoch 108/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5361 - accuracy: 0.8609\n","Epoch 109/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5307 - accuracy: 0.8602\n","Epoch 110/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5340 - accuracy: 0.8609\n","Epoch 111/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5585 - accuracy: 0.8509\n","Epoch 112/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5479 - accuracy: 0.8523\n","Epoch 113/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5464 - accuracy: 0.8602\n","Epoch 114/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5213 - accuracy: 0.8645\n","Epoch 115/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5173 - accuracy: 0.8659\n","Epoch 116/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5255 - accuracy: 0.8616\n","Epoch 117/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5543 - accuracy: 0.8487\n","Epoch 118/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5080 - accuracy: 0.8659\n","Epoch 119/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5068 - accuracy: 0.8652\n","Epoch 120/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5127 - accuracy: 0.8652\n","Epoch 121/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4957 - accuracy: 0.8710\n","Epoch 122/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4952 - accuracy: 0.8731\n","Epoch 123/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5098 - accuracy: 0.8645\n","Epoch 124/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5217 - accuracy: 0.8595\n","Epoch 125/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.5304 - accuracy: 0.8602\n","Epoch 126/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5164 - accuracy: 0.8645\n","Epoch 127/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4974 - accuracy: 0.8667\n","Epoch 128/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4827 - accuracy: 0.8724\n","Epoch 129/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5028 - accuracy: 0.8681\n","Epoch 130/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4839 - accuracy: 0.8746\n","Epoch 131/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4657 - accuracy: 0.8774\n","Epoch 132/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4716 - accuracy: 0.8724\n","Epoch 133/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4875 - accuracy: 0.8695\n","Epoch 134/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4957 - accuracy: 0.8681\n","Epoch 135/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4767 - accuracy: 0.8760\n","Epoch 136/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4591 - accuracy: 0.8810\n","Epoch 137/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4769 - accuracy: 0.8803\n","Epoch 138/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4785 - accuracy: 0.8746\n","Epoch 139/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4553 - accuracy: 0.8789\n","Epoch 140/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4482 - accuracy: 0.8824\n","Epoch 141/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4479 - accuracy: 0.8803\n","Epoch 142/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4360 - accuracy: 0.8853\n","Epoch 143/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4637 - accuracy: 0.8774\n","Epoch 144/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4533 - accuracy: 0.8781\n","Epoch 145/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4378 - accuracy: 0.8767\n","Epoch 146/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4437 - accuracy: 0.8796\n","Epoch 147/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4225 - accuracy: 0.8889\n","Epoch 148/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4464 - accuracy: 0.8810\n","Epoch 149/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4323 - accuracy: 0.8803\n","Epoch 150/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4283 - accuracy: 0.8839\n","Epoch 151/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4195 - accuracy: 0.8889\n","Epoch 152/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4436 - accuracy: 0.8846\n","Epoch 153/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4235 - accuracy: 0.8867\n","Epoch 154/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4155 - accuracy: 0.8896\n","Epoch 155/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4083 - accuracy: 0.8867\n","Epoch 156/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4229 - accuracy: 0.8853\n","Epoch 157/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4096 - accuracy: 0.8918\n","Epoch 158/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3925 - accuracy: 0.8961\n","Epoch 159/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4357 - accuracy: 0.8810\n","Epoch 160/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4458 - accuracy: 0.8781\n","Epoch 161/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4191 - accuracy: 0.8839\n","Epoch 162/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.4015 - accuracy: 0.8918\n","Epoch 163/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3862 - accuracy: 0.9004\n","Epoch 164/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3870 - accuracy: 0.9004\n","Epoch 165/200\n","28/28 [==============================] - 1s 24ms/step - loss: 0.3955 - accuracy: 0.8925\n","Epoch 166/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3774 - accuracy: 0.8982\n","Epoch 167/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3745 - accuracy: 0.8975\n","Epoch 168/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3927 - accuracy: 0.8925\n","Epoch 169/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3990 - accuracy: 0.8946\n","Epoch 170/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3996 - accuracy: 0.8910\n","Epoch 171/200\n","28/28 [==============================] - 1s 24ms/step - loss: 0.3825 - accuracy: 0.8946\n","Epoch 172/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3846 - accuracy: 0.8910\n","Epoch 173/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3940 - accuracy: 0.8968\n","Epoch 174/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3716 - accuracy: 0.9018\n","Epoch 175/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3655 - accuracy: 0.9039\n","Epoch 176/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3785 - accuracy: 0.8989\n","Epoch 177/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3760 - accuracy: 0.8975\n","Epoch 178/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3862 - accuracy: 0.8925\n","Epoch 179/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3615 - accuracy: 0.9032\n","Epoch 180/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3636 - accuracy: 0.9018\n","Epoch 181/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3493 - accuracy: 0.9082\n","Epoch 182/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3707 - accuracy: 0.9032\n","Epoch 183/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3483 - accuracy: 0.9075\n","Epoch 184/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3380 - accuracy: 0.9068\n","Epoch 185/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3308 - accuracy: 0.9068\n","Epoch 186/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3393 - accuracy: 0.9090\n","Epoch 187/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3652 - accuracy: 0.8982\n","Epoch 188/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.5218 - accuracy: 0.8581\n","Epoch 189/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4759 - accuracy: 0.8717\n","Epoch 190/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.4149 - accuracy: 0.8918\n","Epoch 191/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3717 - accuracy: 0.8975\n","Epoch 192/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3917 - accuracy: 0.8946\n","Epoch 193/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3846 - accuracy: 0.8968\n","Epoch 194/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3654 - accuracy: 0.8982\n","Epoch 195/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3339 - accuracy: 0.9125\n","Epoch 196/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3225 - accuracy: 0.9140\n","Epoch 197/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3345 - accuracy: 0.9068\n","Epoch 198/200\n","28/28 [==============================] - 1s 22ms/step - loss: 0.3204 - accuracy: 0.9140\n","Epoch 199/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3322 - accuracy: 0.9082\n","Epoch 200/200\n","28/28 [==============================] - 1s 23ms/step - loss: 0.3231 - accuracy: 0.9133\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7fbe4d18fcf8>"]},"metadata":{"tags":[]},"execution_count":24}]},{"cell_type":"code","metadata":{"id":"6JZgqCRGzJcI","executionInfo":{"status":"ok","timestamp":1604888680998,"user_tz":420,"elapsed":134951,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"89fb7d61-335b-4a27-a570-f88d7d657c9e","colab":{"base_uri":"https://localhost:8080/"}},"source":["y_pred = model.predict(X_test)"],"execution_count":25,"outputs":[{"output_type":"stream","text":["12/12 [==============================] - 0s 7ms/step\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"30zEYM6bzJcK","executionInfo":{"status":"ok","timestamp":1604888680999,"user_tz":420,"elapsed":134946,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"06db7049-9e7e-485f-bdde-94e69fe84cd0","colab":{"base_uri":"https://localhost:8080/"}},"source":["# accuracy improves by adopting LSTM structure\n","print(accuracy_score(y_pred, y_test_))"],"execution_count":26,"outputs":[{"output_type":"stream","text":["0.8464106844741235\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"8pYDluUmzJcM"},"source":["## 4. Stacked LSTM\n","- LSTM layers can be stacked as well"]},{"cell_type":"code","metadata":{"id":"JYecsWDOzJcN","executionInfo":{"status":"ok","timestamp":1604888681000,"user_tz":420,"elapsed":134942,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["def stacked_lstm():\n","    model = Sequential()\n","    model.add(LSTM(50, input_shape = (49,1), return_sequences = True))\n","    model.add(LSTM(50, return_sequences = False))\n","    model.add(Dense(46))\n","    model.add(Activation('softmax'))\n","    \n","    adam = optimizers.Adam(lr = 0.001)\n","    model.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['accuracy'])\n","    \n","    return model"],"execution_count":27,"outputs":[]},{"cell_type":"code","metadata":{"id":"vHjt-dHTzJcP","executionInfo":{"status":"ok","timestamp":1604888681001,"user_tz":420,"elapsed":134939,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["model = KerasClassifier(build_fn = stacked_lstm, epochs = 200, batch_size = 50, verbose = 1)"],"execution_count":28,"outputs":[]},{"cell_type":"code","metadata":{"id":"tDk3d2EMzJcR","executionInfo":{"status":"ok","timestamp":1604888971171,"user_tz":420,"elapsed":425096,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"3427f814-8c09-4a46-ea4f-3065bc964ede","colab":{"base_uri":"https://localhost:8080/"}},"source":["model.fit(X_train, y_train)"],"execution_count":29,"outputs":[{"output_type":"stream","text":["Epoch 1/200\n","28/28 [==============================] - 1s 49ms/step - loss: 2.6792 - accuracy: 0.6437\n","Epoch 2/200\n","28/28 [==============================] - 1s 47ms/step - loss: 1.2435 - accuracy: 0.7147\n","Epoch 3/200\n","28/28 [==============================] - 1s 46ms/step - loss: 1.1688 - accuracy: 0.7147\n","Epoch 4/200\n","28/28 [==============================] - 1s 47ms/step - loss: 1.1525 - accuracy: 0.7147\n","Epoch 5/200\n","28/28 [==============================] - 1s 48ms/step - loss: 1.1345 - accuracy: 0.7147\n","Epoch 6/200\n","28/28 [==============================] - 1s 48ms/step - loss: 1.1144 - accuracy: 0.7147\n","Epoch 7/200\n","28/28 [==============================] - 1s 47ms/step - loss: 1.1124 - accuracy: 0.7147\n","Epoch 8/200\n","28/28 [==============================] - 1s 47ms/step - loss: 1.0941 - accuracy: 0.7147\n","Epoch 9/200\n","28/28 [==============================] - 1s 48ms/step - loss: 1.0806 - accuracy: 0.7147\n","Epoch 10/200\n","28/28 [==============================] - 1s 47ms/step - loss: 1.0536 - accuracy: 0.7147\n","Epoch 11/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.9370 - accuracy: 0.7613\n","Epoch 12/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.8421 - accuracy: 0.7950\n","Epoch 13/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.8718 - accuracy: 0.7907\n","Epoch 14/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.8349 - accuracy: 0.8029\n","Epoch 15/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.8013 - accuracy: 0.8057\n","Epoch 16/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.8083 - accuracy: 0.8057\n","Epoch 17/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7891 - accuracy: 0.8108\n","Epoch 18/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7819 - accuracy: 0.8108\n","Epoch 19/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7865 - accuracy: 0.8143\n","Epoch 20/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.8187 - accuracy: 0.8093\n","Epoch 21/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.8282 - accuracy: 0.8022\n","Epoch 22/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7986 - accuracy: 0.8151\n","Epoch 23/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7882 - accuracy: 0.8122\n","Epoch 24/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7601 - accuracy: 0.8215\n","Epoch 25/200\n","28/28 [==============================] - 1s 46ms/step - loss: 0.7810 - accuracy: 0.8244\n","Epoch 26/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7598 - accuracy: 0.8229\n","Epoch 27/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7476 - accuracy: 0.8237\n","Epoch 28/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7425 - accuracy: 0.8258\n","Epoch 29/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7448 - accuracy: 0.8222\n","Epoch 30/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7413 - accuracy: 0.8272\n","Epoch 31/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7332 - accuracy: 0.8272\n","Epoch 32/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7334 - accuracy: 0.8323\n","Epoch 33/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.7285 - accuracy: 0.8265\n","Epoch 34/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7362 - accuracy: 0.8315\n","Epoch 35/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7350 - accuracy: 0.8258\n","Epoch 36/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7209 - accuracy: 0.8287\n","Epoch 37/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7199 - accuracy: 0.8265\n","Epoch 38/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7197 - accuracy: 0.8294\n","Epoch 39/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.7048 - accuracy: 0.8337\n","Epoch 40/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7085 - accuracy: 0.8308\n","Epoch 41/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7140 - accuracy: 0.8308\n","Epoch 42/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7048 - accuracy: 0.8301\n","Epoch 43/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7033 - accuracy: 0.8301\n","Epoch 44/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.6954 - accuracy: 0.8323\n","Epoch 45/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6975 - accuracy: 0.8280\n","Epoch 46/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6971 - accuracy: 0.8308\n","Epoch 47/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7486 - accuracy: 0.8308\n","Epoch 48/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7459 - accuracy: 0.8323\n","Epoch 49/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7238 - accuracy: 0.8301\n","Epoch 50/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7343 - accuracy: 0.8215\n","Epoch 51/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7437 - accuracy: 0.8244\n","Epoch 52/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7151 - accuracy: 0.8337\n","Epoch 53/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7155 - accuracy: 0.8337\n","Epoch 54/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.7174 - accuracy: 0.8287\n","Epoch 55/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.6976 - accuracy: 0.8330\n","Epoch 56/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.7113 - accuracy: 0.8323\n","Epoch 57/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6859 - accuracy: 0.8337\n","Epoch 58/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.7052 - accuracy: 0.8308\n","Epoch 59/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.6821 - accuracy: 0.8330\n","Epoch 60/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6870 - accuracy: 0.8344\n","Epoch 61/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6757 - accuracy: 0.8366\n","Epoch 62/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6824 - accuracy: 0.8358\n","Epoch 63/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6744 - accuracy: 0.8373\n","Epoch 64/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6704 - accuracy: 0.8387\n","Epoch 65/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.6843 - accuracy: 0.8294\n","Epoch 66/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.6708 - accuracy: 0.8330\n","Epoch 67/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6609 - accuracy: 0.8366\n","Epoch 68/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6570 - accuracy: 0.8373\n","Epoch 69/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6490 - accuracy: 0.8387\n","Epoch 70/200\n","28/28 [==============================] - 1s 46ms/step - loss: 0.6430 - accuracy: 0.8373\n","Epoch 71/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.6590 - accuracy: 0.8358\n","Epoch 72/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6499 - accuracy: 0.8366\n","Epoch 73/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.6899 - accuracy: 0.8287\n","Epoch 74/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6828 - accuracy: 0.8315\n","Epoch 75/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6566 - accuracy: 0.8337\n","Epoch 76/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6415 - accuracy: 0.8358\n","Epoch 77/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6425 - accuracy: 0.8351\n","Epoch 78/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6503 - accuracy: 0.8315\n","Epoch 79/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6290 - accuracy: 0.8373\n","Epoch 80/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6225 - accuracy: 0.8409\n","Epoch 81/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6179 - accuracy: 0.8409\n","Epoch 82/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.6321 - accuracy: 0.8394\n","Epoch 83/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.6161 - accuracy: 0.8423\n","Epoch 84/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6145 - accuracy: 0.8401\n","Epoch 85/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6131 - accuracy: 0.8394\n","Epoch 86/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6109 - accuracy: 0.8423\n","Epoch 87/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6245 - accuracy: 0.8401\n","Epoch 88/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.6004 - accuracy: 0.8452\n","Epoch 89/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.5906 - accuracy: 0.8473\n","Epoch 90/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.5969 - accuracy: 0.8459\n","Epoch 91/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.5843 - accuracy: 0.8459\n","Epoch 92/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.5762 - accuracy: 0.8495\n","Epoch 93/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.5896 - accuracy: 0.8459\n","Epoch 94/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.5742 - accuracy: 0.8473\n","Epoch 95/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.5711 - accuracy: 0.8487\n","Epoch 96/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.5767 - accuracy: 0.8473\n","Epoch 97/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.5627 - accuracy: 0.8516\n","Epoch 98/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.5740 - accuracy: 0.8473\n","Epoch 99/200\n","28/28 [==============================] - 1s 46ms/step - loss: 0.5713 - accuracy: 0.8444\n","Epoch 100/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.5533 - accuracy: 0.8487\n","Epoch 101/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.5494 - accuracy: 0.8538\n","Epoch 102/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.5462 - accuracy: 0.8516\n","Epoch 103/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.5502 - accuracy: 0.8495\n","Epoch 104/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.5367 - accuracy: 0.8523\n","Epoch 105/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.5585 - accuracy: 0.8502\n","Epoch 106/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.5431 - accuracy: 0.8523\n","Epoch 107/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.5478 - accuracy: 0.8495\n","Epoch 108/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.5306 - accuracy: 0.8566\n","Epoch 109/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.5389 - accuracy: 0.8502\n","Epoch 110/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.5170 - accuracy: 0.8609\n","Epoch 111/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.5358 - accuracy: 0.8523\n","Epoch 112/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.5196 - accuracy: 0.8638\n","Epoch 113/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.5063 - accuracy: 0.8695\n","Epoch 114/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.4982 - accuracy: 0.8602\n","Epoch 115/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.5141 - accuracy: 0.8602\n","Epoch 116/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.5002 - accuracy: 0.8616\n","Epoch 117/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.5126 - accuracy: 0.8631\n","Epoch 118/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.4830 - accuracy: 0.8645\n","Epoch 119/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.4710 - accuracy: 0.8746\n","Epoch 120/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.4659 - accuracy: 0.8760\n","Epoch 121/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.4698 - accuracy: 0.8746\n","Epoch 122/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.4564 - accuracy: 0.8810\n","Epoch 123/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.4509 - accuracy: 0.8796\n","Epoch 124/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.4433 - accuracy: 0.8824\n","Epoch 125/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.4362 - accuracy: 0.8875\n","Epoch 126/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.4422 - accuracy: 0.8853\n","Epoch 127/200\n","28/28 [==============================] - 1s 52ms/step - loss: 0.4220 - accuracy: 0.8889\n","Epoch 128/200\n","28/28 [==============================] - 2s 63ms/step - loss: 0.4217 - accuracy: 0.8939\n","Epoch 129/200\n","28/28 [==============================] - 2s 82ms/step - loss: 0.4118 - accuracy: 0.8896\n","Epoch 130/200\n","28/28 [==============================] - 2s 80ms/step - loss: 0.4169 - accuracy: 0.8875\n","Epoch 131/200\n","28/28 [==============================] - 2s 80ms/step - loss: 0.4244 - accuracy: 0.8867\n","Epoch 132/200\n","28/28 [==============================] - 2s 82ms/step - loss: 0.4258 - accuracy: 0.8853\n","Epoch 133/200\n","28/28 [==============================] - 2s 82ms/step - loss: 0.4232 - accuracy: 0.8875\n","Epoch 134/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.4126 - accuracy: 0.8882\n","Epoch 135/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.4003 - accuracy: 0.8953\n","Epoch 136/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.3858 - accuracy: 0.8968\n","Epoch 137/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.3916 - accuracy: 0.8953\n","Epoch 138/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.3863 - accuracy: 0.8975\n","Epoch 139/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.3826 - accuracy: 0.8968\n","Epoch 140/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3799 - accuracy: 0.8953\n","Epoch 141/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.3772 - accuracy: 0.9018\n","Epoch 142/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.3748 - accuracy: 0.9011\n","Epoch 143/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.3693 - accuracy: 0.9025\n","Epoch 144/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.3581 - accuracy: 0.9054\n","Epoch 145/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.4102 - accuracy: 0.8867\n","Epoch 146/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3920 - accuracy: 0.8946\n","Epoch 147/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3567 - accuracy: 0.9032\n","Epoch 148/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.3627 - accuracy: 0.9025\n","Epoch 149/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3504 - accuracy: 0.9032\n","Epoch 150/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3403 - accuracy: 0.9082\n","Epoch 151/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3405 - accuracy: 0.9075\n","Epoch 152/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.3499 - accuracy: 0.9104\n","Epoch 153/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3392 - accuracy: 0.9054\n","Epoch 154/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.3205 - accuracy: 0.9140\n","Epoch 155/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.3142 - accuracy: 0.9147\n","Epoch 156/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3178 - accuracy: 0.9111\n","Epoch 157/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3180 - accuracy: 0.9140\n","Epoch 158/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3177 - accuracy: 0.9118\n","Epoch 159/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.3106 - accuracy: 0.9168\n","Epoch 160/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.3041 - accuracy: 0.9161\n","Epoch 161/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3097 - accuracy: 0.9183\n","Epoch 162/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3047 - accuracy: 0.9161\n","Epoch 163/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.2939 - accuracy: 0.9240\n","Epoch 164/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.3197 - accuracy: 0.9183\n","Epoch 165/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.3179 - accuracy: 0.9133\n","Epoch 166/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3151 - accuracy: 0.9111\n","Epoch 167/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.3003 - accuracy: 0.9161\n","Epoch 168/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3169 - accuracy: 0.9090\n","Epoch 169/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3407 - accuracy: 0.9061\n","Epoch 170/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.3023 - accuracy: 0.9147\n","Epoch 171/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2891 - accuracy: 0.9226\n","Epoch 172/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.2886 - accuracy: 0.9247\n","Epoch 173/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.2910 - accuracy: 0.9154\n","Epoch 174/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.3099 - accuracy: 0.9161\n","Epoch 175/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.2992 - accuracy: 0.9204\n","Epoch 176/200\n","28/28 [==============================] - 1s 51ms/step - loss: 0.2837 - accuracy: 0.9197\n","Epoch 177/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.2744 - accuracy: 0.9211\n","Epoch 178/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.2654 - accuracy: 0.9283\n","Epoch 179/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.2610 - accuracy: 0.9305\n","Epoch 180/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.2759 - accuracy: 0.9226\n","Epoch 181/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2899 - accuracy: 0.9190\n","Epoch 182/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.2952 - accuracy: 0.9140\n","Epoch 183/200\n","28/28 [==============================] - 1s 51ms/step - loss: 0.2693 - accuracy: 0.9262\n","Epoch 184/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2648 - accuracy: 0.9262\n","Epoch 185/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2747 - accuracy: 0.9276\n","Epoch 186/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.2641 - accuracy: 0.9247\n","Epoch 187/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2544 - accuracy: 0.9283\n","Epoch 188/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.2399 - accuracy: 0.9341\n","Epoch 189/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2413 - accuracy: 0.9384\n","Epoch 190/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2706 - accuracy: 0.9226\n","Epoch 191/200\n","28/28 [==============================] - 1s 48ms/step - loss: 0.2550 - accuracy: 0.9283\n","Epoch 192/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2402 - accuracy: 0.9312\n","Epoch 193/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2337 - accuracy: 0.9348\n","Epoch 194/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.2311 - accuracy: 0.9376\n","Epoch 195/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2246 - accuracy: 0.9376\n","Epoch 196/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2343 - accuracy: 0.9333\n","Epoch 197/200\n","28/28 [==============================] - 1s 51ms/step - loss: 0.4001 - accuracy: 0.8860\n","Epoch 198/200\n","28/28 [==============================] - 1s 47ms/step - loss: 0.3215 - accuracy: 0.9140\n","Epoch 199/200\n","28/28 [==============================] - 1s 49ms/step - loss: 0.2765 - accuracy: 0.9211\n","Epoch 200/200\n","28/28 [==============================] - 1s 50ms/step - loss: 0.2582 - accuracy: 0.9276\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7fbe4ca364e0>"]},"metadata":{"tags":[]},"execution_count":29}]},{"cell_type":"code","metadata":{"id":"0JU5eZ1tzJcT","executionInfo":{"status":"ok","timestamp":1604888971870,"user_tz":420,"elapsed":425789,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"2b84607f-3816-450f-c14c-78ceeccb31d8","colab":{"base_uri":"https://localhost:8080/"}},"source":["y_pred = model.predict(X_test)"],"execution_count":30,"outputs":[{"output_type":"stream","text":["12/12 [==============================] - 0s 13ms/step\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"I9la6CXyzJcV","executionInfo":{"status":"ok","timestamp":1604888972100,"user_tz":420,"elapsed":426014,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"7018f256-800f-4f28-f437-5fb822b585a0","colab":{"base_uri":"https://localhost:8080/"}},"source":["print(accuracy_score(y_pred, y_test_))"],"execution_count":31,"outputs":[{"output_type":"stream","text":["0.8480801335559266\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"8jxeWrY_zZ3K"},"source":[""],"execution_count":null,"outputs":[]}]}